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ARTICLE 

Adapting a MOOC for Research: Lessons Learned from 
the First Presentation of Literature and Mental Health: 
Reading for Wellbeing 

Rachael Hodge 


The University of Warwick's FutureLearn MOOC Literature and Mental Health: Reading for Wellbeing, 
which began its first presentation February 2016, was identified as an opportunity to conduct some 
research into the course subject area, 'reading for wellbeing' or 'bibliotherapy'. Since 2013, a substantial 
body of literature has emerged in the field of MOOC-related research, with the MOOC becoming both 
the subject of and vehicle for research. The research approach adopted in Literature and Mental Health 
was influenced by other, recent research studies conducted within MOOCs, and particularly by the first 
presentation of Monash University's Mindfulness for Wellbeing and Peak Performance FutureLearn MOOC, 
which distributed a stress survey to its learners in the first and final weeks of the course, to assess the 
efficacy of the course's mindfulness practices. 

A number of reasons for trialling the use of this MOOC as a research tool were identified at the pro¬ 
ject's outset. MOOCs give researchers access to large numbers of possible research participants, making 
MOOC research an attractive prospect, while the opportunity to gather valuable, potentially publishable 
data from free online courses may help to justify the time and resources expended during the production 
of new MOOCs. Several additional benefits of in-MOOC research were discovered during the process, 
including the potential for research activities to enrich the learner experience. Flowever, a number of 
challenges and limitations were also encountered during the development of the study; the inevitable self¬ 
selection bias among MOOC learners, and the difficulty of establishing a control group within the MOOC 
activities, posed impediments to the gathering of useful, publishable data. 

Although we were aware of other MOOCs which had been used as vehicles for research, the process 
of adapting Literature and Mental Health for this research study was nonetheless an illuminating and 
instructive experience. The purpose of this paper is to reflect on that experience, and to consider the 
lessons learned during the process which may be useful in informing future research studies conducted 
via Massive Open Online Courses. 
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Introduction 

The University of Warwick’s FutureLearn Massive Open 
Online Course (MOOC) Literature and Mental Health: 
Reading for Wellbeing began its first presentation in Feb¬ 
ruary 2016 (University of Warwick, 2016). In the early 
stages of its development, this MOOC had been identified 
as an opportunity to conduct some research into the sub¬ 
ject area, using the MOOC participants to investigate the 
relationship between reading and wellbeing. 

Literature and Mental Health: Reading for Wellbeing is a 
six-week MOOC, with each of its weeks organised around 
a different mental health condition; the first week concen¬ 
trates on ‘Stress’, for instance, while the fifth is concerned 


University of Warwick, GB 
Rachael,hodge@wbs.ac.uk 


with ‘Depression and Bipolar Disorder’. The course con¬ 
tent is broadly divisible into two strands: the first is to 
explore how different mental health conditions have been 
written about in literature through the ages; the second is 
to consider the extent to which reading might have some 
therapeutic value in the treatment and prevention of cer¬ 
tain mental health conditions. It was the latter of these 
strands which our research was designed to explore. 

This paper will begin by putting the Literature and 
Mental Health research study into context, explaining 
the rationale behind the decision to conduct research 
via a MOOC, giving a brief overview of the history of 
MOOC research so far, and describing the research 
activities designed for Literature and Mental Health. 
The main body of the paper will be occupied with a 
discussion of the lessons learned about MOOC research 
during the first presentation of Literature and Mental 
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Health, and will be organised under five headings: 
research participation rates; the problem of self-selec¬ 
tion bias; balancing research and learning; impact find¬ 
ings; and the value of qualitative data. 

The field of MOOC research is both relatively new and 
rapidly expanding. As such, it is important to reflect on the 
process of researching via a MOOC, as well as on the find¬ 
ings of the research conducted. We hope that our reflec¬ 
tions on the Literature and Mental Health research project 
will encourage, influence and inform future research con¬ 
ducted via MOOCs. 

The Research in Context 

Background: rationalising the research study 

The Literature and Mental Health MOOC was developed by 
the University of Warwick in conjunction with the organi¬ 
sation ReLit: The Bibliotherapy Foundation. A number of 
reasons for conducting a research study via this MOOC 
were identified at the project’s outset, some applicable 
to MOOCs in general, and others specific to the content 
of Literature and Mental Health. For any university, the 
prospect of generating publishable research via a MOOC 
may help to justify the time and resources expended 
during MOOC production; in the case of this particular 
MOOC, meanwhile, the opportunity to conduct large- 
scale research into the bibliotherapy practices advocated 
by ReLit was too valuable to be overlooked. 

In 2012, Thomas L. Friedman heralded a higher educa¬ 
tion ‘revolution’ in his New York Times article about the 
newly developed Coursera MOOC platform (Friedman, 
2012). The Massive Open Online Course, he suggested, 
was an innovation that would allow thousands of people 
to overcome the financial and geographical barriers that 
had previously prevented them from accessing university- 
level education (Friedman, 2012). These predictions of a 
bright and momentous future for MOOCs were inevita¬ 
bly followed by a backlash, with critics pointing to low 
completion rates and the limited demographic diversity 
of learners as evidence of the failure of MOOCs to revo¬ 
lutionise higher education (Zemsky, 2014). Yet although 
Friedman’s vision of an egalitarian online future for uni¬ 
versity learning has not been realised, nor can the MOOC 
be dismissed as an abject failure: MOOCs are, for the time 
being, here to stay (Friedman, 2013; Matkin, 2015). 

Nonetheless, the novelty of the Massive Open Online 
Course for participating universities is diminishing, and 
the MOOC business model is under increasing scrutiny. 
A study in the Times Higher Education estimated that the 
average cost, for a university, of developing a MOOC was 
£29,356 in 2015, based on a survey of twenty courses 
from nine UK institutions (Parr, 2015). Although some 
universities and platform providers have found ways to 
profit from MOOCs (see, for example, Custer, 2014), many 
MOOCs yield negligible direct revenue, and so higher edu¬ 
cation institutions must look for other ways to make these 
online courses pay. One solution is to reuse MOOC teach¬ 
ing materials within paid courses. Another is to develop 
MOOCs tailored to attract students to particular degree 
courses offered by an institution. Some platforms and insti¬ 
tutions have also started to offer MOOCs for credit, giving 


learners the opportunity to pay for and pass an additional 
assessment following the completion of a MOOC, confer¬ 
ring academic credit that can count towards a particular 
qualification or degree (FutureLearn, 2016). For universi¬ 
ties, research output is another, potential form of revenue 
to be gained from MOOCs. This is an important reason for 
trialling MOOC research projects generally. 

The reasons for conducting a research study via 
Literature and Mental Health, in particular, are related 
to the primary initiative of the organisation ReLit. The 
therapeutic reading advocated by ReLit is a modern and 
progressive approach in the field of mental health treat¬ 
ment, but the concept of bibliotherapy - or ‘reading for 
wellbeing’ - is rooted in ancient ideas. The entrance to the 
library of Pharaoh Ramses II reportedly bore an inscrip¬ 
tion declaring the building a ‘house of healing for the 
soul’ (Lutz, 1978, p. 36) and in the fifth century BC, the 
Greek tragedian Aeschylus wrote that Words are the phy¬ 
sicians of the mind diseased' (Kelly, 2014, p. 74). Despite 
boasting an anecdotal history spanning over 2000 years, 
however, the medical practice of bibliotherapy is sup¬ 
ported by little substantive evidence. Some research stud¬ 
ies investigating the therapeutic potential of literature in 
different clinical settings have been conducted in recent 
years (see, for example, Gregory et al., 2004; Moldovan 
et al., 2013), but as yet, none of these trials has yielded 
results significant enough to elevate bibliotherapy to the 
status of mindfulness, another form of non-pharmaceuti- 
cal treatment proven to be ‘as good as drugs for prevent¬ 
ing depression relapse’, and given its own page on the 
UK’s National Health Service website (NHS Choices, 2016). 
There exists, therefore, an incentive to gather data rele¬ 
vant to the potential link between reading and wellbeing 
at any available opportunity. The Literature and Mental 
Health: Reading for Wellbeing MOOC presented itself as 
one such opportunity. 

MOOC research: the story so far 

Veletsianos and Shepherdson observe that a ‘deluge of 
empirical research’ concerning MOOCs has emerged since 
2013 (Veletsianos & Shepherdson, 2016, p.198). In their 
literature review of the MOOC-related literature reviews 
published since this date, Veletsianos and Shepherdson 
identify four dominant themes of MOOC research publi¬ 
cations: the distinction between cMOOCs and xMOOCs; 
the impact of MOOCs on education and pedagogical prac¬ 
tices; the demographic of MOOC users; and the challenges 
encountered by MOOCs in the initiative’s infancy (Veletsi¬ 
anos & Shepherdson, 2016). Among the research propos¬ 
als submitted to the MOOC Research Initiative, funded 
by the Gates Foundation and administered by Athabasca 
University in 2013, meanwhile, nine research themes were 
identified, including learner performance, MOOC plat¬ 
forms, MOOC-led improvements in educational technol¬ 
ogy, and the impact of MOOCs on higher education insti¬ 
tutions (Gasevic et al., 2014). 

As well as providing researchers with a new object of study, 
the MOOC has also encouraged the development of new 
research approaches: researchers are, increasingly, think¬ 
ing through and with MOOCs, as well as looking at them. 
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Admittedly, the distinction between research conducted on 
MOOCs and research conducted through MOOCs is not an 
easy one to draw, but for the purposes of this paper, research 
conducted through MOOCs will be defined as any research 
involving a MOOC, in which the findings have an applica¬ 
tion beyond that MOOC or MOOCs in general. Thus, papers 
which seek to identify and describe the differences between 
xMOOCs and cMOOCs (see, for example, Daniels, 2012; 
Siemens, 2012) are concerned with research on MOOCs, 
self-reflexive research that looks at MOOCs to determine 
something about MOOCs. The 2015 study by Southampton 
University, which analysed MOOC learner comments in 
order to evaluate different indicators of pedagogical activity, 
meanwhile, is an example of research conducted through a 
MOOC: although MOOC learner comments were the sub¬ 
ject of the analysis, the aim of this research study was to 
compare different measures of pedagogical validity, and the 
findings could be applied to any online learning situation 
(O'Riordan et ah, 2015). 

The research project designed for and included in 
Literature and Mental Health seems to exist in a slightly 
different category again, in that the MOOC was used as 
a vehicle for research into a subject area entirely unre¬ 
lated to MOOCs in general. As this paper will discuss, we 
have inevitably made some important findings about the 
impact and potential of the MOOC platform, but the aim 
of the research was to explore the relationship between 
reading and wellbeing, with the MOOC simply providing 
a convenient opportunity to do so. Other MOOCs have 
attempted similar research studies, though not neces¬ 
sarily on the same scale. The first presentation of the 
Mindfulness for Wellbeing and Peak Performance MOOC, 
produced by Monash University and hosted on the 
FutureLearn MOOC platform, gave learners the oppor¬ 
tunity to take part in a ‘stress survey’ at the beginning 
and end of the course. This survey permitted learners 
to observe whether the mindfulness practices they had 
learned during the course were helping to reduce their 
stress levels, but learners were also given the oppor¬ 
tunity to make their survey responses available to the 
lead educators, to be used as part of a research study 
(Monash University, 2015). The University of Warwick’s 
first MOOC, The Mind is Flat, meanwhile, invited learners 
to take part in a short psychological experiment related 
to the course content in each of its six weeks (University 
of Warwick, 2013). These experiments were included pri¬ 
marily to give learners an insight into the workings of 
the mind and into the psychological experimentation 
process, but the results could also be used as part of a 
research study. 

It is notable, and perhaps not entirely coincidental, 
that the three in-MOOC research projects described here, 
which look beyond the MOOC to investigate a particular 
subject area, are all concerned with human psychology. 
Certain topics will naturally lend themselves more than 
others to in-MOOC research, and the remote access that 
MOOCs give researchers to human research participants 
seems suited to online experiments exploring aspects of 
psychology. The use of MOOC learners as research par¬ 
ticipants - and as research participants in a psychological 


experiment, in particular - raises a number of ethical 
questions, however, which are addressed below. 

Although researchers are only beginning to use MOOCs 
as opportunities to involve their participants in these kinds 
of surveys and experiments, the use of internet-based 
platforms for large scale data-gathering is, in 2016, very 
well established. Survey design websites such as Qualtrics 
(which we used to build the surveys for Literature and 
Mental Health) and SurveyMonkey make it easy for any¬ 
one to distribute research surveys via email, or to embed 
them as links in webpages. Retailers regularly redirect 
online customers to questionnaires designed to gather 
information about their shopping habits and experiences. 
An initiative such as the National Student Survey utilises 
internet-based research to gather data on a national scale. 
To take another example, this time relevant to the topics 
explored in Literature and Mental Health, the University 
of Cambridge’s ‘Poetry and Memory' project is using an 
online survey, accessed on the project’s website www. 
poetryandmemory.com, to ‘investigate] the value and 
experience of poetry in the memory, and examin[e] the 
relationship between memorisation and understanding’ 
(The Poetry and Memory Project, 2014). This last exam¬ 
ple, in particular, suggests that internet-based research 
methods are being used not only to gather large amounts 
of data, but also to investigate, in some depth, questions 
of a cognitive nature. The MOOC platform is primarily 
pedagogical, and MOOC research is therefore subject to 
more restrictions than other research conducted via the 
Internet; this is an issue discussed in more detail below. 
Nevertheless, there is certainly no lack of precedent or 
model for online data gathering, and it seems that MOOC 
developers could exploit the opportunity to conduct 
online research more readily than they currently do. 

A note on MOOC research ethics 

In his comprehensive discussion of MOOC ethics, Stephen 
Marshall alludes to a number of the challenges facing in- 
MOOC research projects, several of which we encountered 
during the development of Literature and Mental Health 
(Marshall, 2013). Issues of data protection are inevitably 
fraught when information is collected and stored online. 
We made the decision to design and conduct all of our 
research activities using the survey platform Qualtrics, 
which sets out its commitment to ‘the protection and reli¬ 
ability of customer data' in its online security statement, 
as well as outlining the practices and regulations in place 
to ensure this data protection (Qualtrics, 2016). 

In addition to the problem of data protection, inter- 
net-based research conducted within MOOCs poses 
further ethical challenges, in relation to learners’ aware¬ 
ness of and consent to their involvement in a research 
study. FutureLearn’s research ethics guidelines remind 
FutureLearn Partners that individuals taking part in 
FutureLearn courses are ‘engaging voluntarily with the 
courses with the intention of learning’ (FutureLearn, 
2014). Learners have not signed up to be research par¬ 
ticipants, and while research on and through MOOCs is 
permitted, FutureLearn Partners have a responsibility 
to make learners aware that their participation in and 
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contributions to a particular course may be monitored 
and analysed for research purposes. 

Presumably for reasons of practicality, the FutureLearn 
research ethics guidelines do not require Partners to seek 
‘opt-in consent’ from participants: provided learners are 
clearly notified of the research taking place, they ‘can 
be assumed to have given consent for participation in 
research’ (FutureLearn, 2014). Given the potentially sen¬ 
sitive nature of our research interests in Literature and 
Mental Health, we were concerned to ensure that learn¬ 
ers knew precisely when they were engaging in a research 
activity, and that they were able to participate in the course 
without contributing to the research if they preferred. The 
research activities were therefore conducted on a separate 
platform to the remainder of the course. After being noti¬ 
fied that their responses may be used for research, learn¬ 
ers had to actively follow a link to launch the relevant 
survey or activity; alternatively, they were able to proceed 
to the next step without participating. Moreover, learners 
were given the option of supplying an email address as 
an identifier for their response only after they had com¬ 
pleted a survey or activity, so that they were fully aware of 
any information divulged prior to supplying an identifier. 
This identifier would then be used to match all survey and 
activity responses created by the same learner, to build up 
a profile for that learner. 

The FutureLearn research ethics guidelines stipulate 
that ‘[participants should be given opportunities to access 
the outcomes of research in which they have participated’ 
(FutureLearn, 2014). In Literature and Mental Health, 
learners were given the opportunity to sign up for research 
updates at the end of the Start of Course survey. Emails 
will be circulated to those learners who asked to be kept 
updated, detailing the results of the course surveys and 
activities as well as outlining any future plans for research. 
The first of these emails, which shared with learners our 
findings about learner reading habits, was distributed on 
23 June 2016, to 2665 recipients. Of these, 1372 or 51.5% 
opened the email, and 253 followed the link to download 
the full, in-depth report on reading habits. 

Given the subject matter of Literature and Mental Health, 
it was deemed necessary to seek further ethical guidance 
on the specific content of the MOOC research activities, 
which falls outside the usual remit of Warwick Business 
School research. All surveys and research activities were 
therefore checked and approved by a clinical psychologist 
prior to their inclusion in the course. 

Research activities: what did we do? 

Our MOOC research project comprised seven surveys and 
activities: 

StartofCoursesurvey(week l,step 1.3):Thissurvey 
asked participants their reasons for taking part in 
Literature and Mental Health: Reading for Wellbe¬ 
ing, establishing their current attitudes towards the 
therapeutic use of literature, and inviting them to 
share details about their reading habits. This survey 
was also designed to gather basic demographic data 
about participants, including age, gender, educational 


background and location, and at the conclusion of 
the survey, participants were given the opportunity 
to volunteer for a post-course online research study, 
which will further investigate the relationship be¬ 
tween reading and wellbeing. 

• Poetry for Stress Relief activity (week 1, step 1.7): In 
this activity, participants were asked to register their 
current stress levels on six sliding scales. Each partici¬ 
pant was then shown, at random, one of five versions 
of the same poem (‘Leisure’, W. H. Davies). Two of 
the versions were text only, one version was an audio 
recording accompanied by onscreen text, one version 
was an audio recording accompanied by onscreen 
images of natural settings, and one was a video re¬ 
cording of the lead educator reading the poem. After 
reading or listening to the poem, learners were asked 
to register their stress levels again, on the same six 
sliding scales. 

• Reading for Wellbeing survey (week 5, step 5.9): this 
short survey invited participants to supply more 
information about their reading habits, and specifi¬ 
cally, about whether and what they read for comfort. 

• Missing Word activity (week 5, step 5.14): partici¬ 
pants were presented with an anonymised version 
of Edward Thomas’ poem ‘Melancholy’, with several 
of its words and its title removed. Participants then 
selected words to fill in these gaps from a dropdown 
selection of three to five words. The purpose of this 
activity was to observe whether learners chose words 
with more positive or more negative connotations. 

• Poetry and Memory activity (week 6, step 6.10): in 
this activity, participants were given a short poem or 
prose extract to read. After spending some time try¬ 
ing to memorise the text, they were shown the same 
text again with some words missing, and were asked 
to try to fill in the words from memory. There were 
up to three extracts for participants to attempt. This 
activity was designed to investigate whether poetry 
is more easily memorable than prose, and whether 
language features such as rhyme and alliteration aid 
memorisation. 

• End of Course survey (week 6, step 6.15): participants 
in the End of Course survey were asked to reflect on 
their experience of the course, and to evaluate the 
course teaching materials, both in terms of content 
and format. 

• Follow-Up survey: this survey was distributed to all 
MOOC learners in an email, ten weeks after the con¬ 
clusion of the course. The questions were designed 
to assess the impact of Literature and Mental Health 
on participants’ online learning habits and reading 
habits. 

All of these surveys and activities were built using the 
survey platform Qualtrics. With the exception of the 
Follow-Up survey, all surveys and activities were accessed 
by learners via links from steps within the course. At the 
end of each survey and activity, learners were invited to 
supply an email address to be used as a unique identi¬ 
fier. This allows us to match survey and activity responses 
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generated by the same individual, building up profiles for 
those learners who have chosen to supply an identifying 
email address. 

In the Start of Course survey, learners were also given the 
opportunity to volunteer to participate in a post-course 
online study. This study is currently in the early stages of 
its development, and will serve as a more extensive and 
rigorous counterpart to the research activities conducted 
within the MOOC itself. 

The MOOC as Research Tool: Lessons Learned 
Research participation rates 

Perhaps the most obvious asset of the MOOC as a research 
tool is the access that it can afford researchers to thou¬ 
sands of potential research participants. The big num¬ 
bers do tend to diminish rather quickly under scrutiny, of 
course. Of the 23,030 people who signed up to Literature 
and Mental Health, for instance, only 12,474 ever accessed 
the course. Nonetheless, with thousands of potential par¬ 
ticipants, these remain large numbers for a research study, 
as a comparison with some recent and influential mind¬ 
fulness trials will demonstrate. Learner numbers typically 
reduce as a MOOC progresses, and participation in the 
Literature and Mental Health research activities inevitably 
dropped concurrently. Even ten weeks after the conclusion 
of the course, however, we found that over 1000 learners 
were willing to take part in a Follow-Up survey distributed 
by email; not only does this indicate that participant num¬ 
bers remained high in comparison to non-MOOC research 
trials, but it also demonstrates that email distribution may 
be more successful than we had originally anticipated. 

The participant numbers for recent clinical trials inves¬ 
tigating the efficacy of mindfulness provide an appo¬ 
site point of comparison, helping to put the participant 
numbers for the Literature and Mental Health research 
activities into some context. Like bibliotherapy, mindful¬ 
ness-based cognitive therapy (MBCT) is a non-pharmaco- 
logical approach to improving and maintaining mental 
wellbeing. Mindfulness has become increasingly popular 
in recent years: schools and universities offer mindful¬ 
ness workshops to their students; businesses are begin¬ 
ning to introduce mindfulness to reduce workplace stress 
and increase productivity; in April 2015, The Mindfulness 
Colouring Book was an Amazon US top ten bestseller 
(Flood, 2015); and mindfulness is now prescribed on the 
NHS, with a section of the NHS website now dedicated 
to the practice (NHS Choices, 2016). Several influential 
publications in recent years have established the status 
of mindfulness as more than simply a popular trend, 
demonstrating that the practice is as effective at prevent¬ 
ing relapse in recurrent depression as anti-depressant 
medication (ADM). 

The clinical trial cited on the NHS website, which found 
MBCT to be an equally effective alternative to ADM for the 
prevention of relapse in recurrent depression, analysed 
424 participants, randomly assigned to two equal groups 
(Kuyken et al., 2015). In 2010, researchers at the Oxford 
Mindfulness Centre published the design and protocol for 
a trial comparing MBCT with a control psychological treat¬ 
ment; allowing for an expected attrition rate of 20%, they 


intended to recruit 375 participants, in order to achieve a 
sample size of at least 300 (Williams et al., 2010). Another 
research project conducted by the Oxford Mindfulness 
Centre, which studied the effectiveness of mindfulness 
in treating health anxiety or hypochondriasis, recruited a 
total of 74 participants. 

Evidently, a MOOC such as Literature and Mental Health: 
Reading for Wellbeing gives researchers access to a cohort 
of potential participants many times larger than the groups 
analysed in the most influential of mindfulness studies. 
During the Start of Course, learners were given the oppor¬ 
tunity to volunteer to take part in a post-course online 
research study, which is currently being developed. Of the 
9754 learners who responded to the survey, 4009 volun¬ 
teered to participate in this post-course online study. This 
number is almost ten times larger than the number of par¬ 
ticipants in the influential 2015 mindfulness trial, although 
it is, of course, impossible to predict how many of these 
volunteers will ultimately take part in the online study. 

Figure 1 shows the number of responses to each course 
activity, compared to the number of active learners partici¬ 
pating in the same week of the course. An active learner 
is defined as a learner who marks at least one step of the 
course as ‘complete’. The graph demonstrates a dramatic 
but not atypical decline in the number of active learners 
participating in the course over the 6 weeks; the number 
of active learners in week 2 is 56.1% of the number in 
week 1, while the number in week 6 is 29.5% of the week 
1 total. The corresponding decline in the number of learn¬ 
ers participating in the course activities over the six weeks 
is therefore inevitable. 

Notably, however, the percentage of active learners 
responding to the activities also diminishes across the six 
weeks, rather than remaining constant. As a percentage 
of the number of active learners in week 1, the number of 
responses to the first course activity (the Start of Course 
survey) is 97.1%. By contrast, the number of responses to 
the End of Course survey (the second activity in week 6) is 
just 62.9% of the number of active learners participating 
in that week. This difference may, in part, be accounted for 
by the fact that the Start of Course survey featured near 
the beginning of a week, in step 1.3. The second activity in 
week 1, the Poetry for Stress Relief activity (step 1.7), had a 
lower response rate of 84.9%, suggesting some attrition in 
participation rates within an individual week. We can iden¬ 
tify this same trend in weeks 5 and 6, with fewer responses 
to the second activity in each week than the first. The End 
of Course survey features in step 6.15, near the conclu¬ 
sion of the week. The Missing Word activity, in week 5, is 
situated in a similar position, in step 5.14. The number of 
responses to this activity is 76.2% of the total number of 
active learners in week 5, lower than the percentage of 
active learners responding to the first activity in week 1, 
but not as low as the percentage of learners responding to 
the second activity in week 6 (62.9%). We see, therefore, 
that this attrition has taken place both within individual 
weeks and across the course as a whole. 

Nonetheless, the numbers of research activity partici¬ 
pants remain higher throughout the course than in any 
of the mindfulness trials discussed above. The number of 
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Figure 1: Graph comparing the number of active learners to the number of activity responses during the first presenta¬ 
tion of Literature and Mental Health: Reading for Wellbeing. 


responses generated by the follow-up survey surprised us, 
moreover. We had made the decision to include all surveys 
and activities within the course itself, as we believed this 
would result in higher levels of participation than if the 
surveys and activities were distributed via email links. We 
were pleased, therefore, that ten weeks after the conclu¬ 
sion of the course, 1339 learners (71.9% of the number 
that responded to the End of Course survey) responded to 
the Follow-Up survey distributed in an email. It should be 
noted that 87.9% of these respondents indicated that they 
had not viewed the course for at least a month. 

Learners had been invited to supply an email address 
at the end of every survey or activity, which would be 
used as a unique identifier, allowing profiles of learner 
responses to be compiled. Figure 2 shows the total num¬ 
ber of responses to each activity, alongside the number of 
responses with email addresses supplied. Figure 3 shows 
the percentage of responses to each activity in which an 
email address was supplied. In five of the seven surveys or 
activities, over 50% of responses included an email address 
to be used as an identifier. 972 email addresses could be 
matched between the Start of Course and End of Course 
surveys; this is 16.3% of the number of email addresses 
supplied in the Start of Course survey, and 79.8% of the 
number of email addresses supplied in the End of Course 
survey. Across all seven surveys and activities, meanwhile, 
157 email addresses could be matched. The sample size 
therefore becomes much smaller if only those learners with 
a full profile of survey and activity responses are consid¬ 
ered. Indeed, this number is smaller than the sample sizes 
of two of the three mindfulness studies referenced above. 

We had anticipated large numbers of research activity 
participants, and the MOOC did not disappoint. Despite 


a drop-off as the course progressed, the majority of active 
learners participated in the research activities in every 
week of the course. Participant numbers remained far 
higher than those for comparable non-MOOC research 
studies, although as this paper discusses below, the 
Literature and Mental Health research exercises lacked the 
selection rigour of the mindfulness trials to which they 
have been compared. Upon reflection, then, participant 
numbers do indeed make MOOC research an attractive 
prospect; the challenge is maintaining and identifying a 
cohort of participants across multiple research activities. 

The problem of self-selection bias 

In a research trial, the size of a sample group, while 
important, is by no means the only factor that needs to 
be taken into consideration. Despite the large numbers of 
responses generated by the Literature and Mental Health 
research activities, it must be acknowledged that there 
exists a self-selection bias among the MOOC learners. 
Every participant in the surveys and activities - indeed, 
every volunteer for the post-course online study - has 
signed up for an educational online course entitled Lit¬ 
erature and Mental Health: Reading for Wellbeing. It seems 
probable, therefore, that these participants would be 
more likely to believe in or, at least, to be receptive to the 
concept of ‘reading for wellbeing’ than a random sample 
of the general population. 

To demonstrate this, we asked learners in the Start of 
Course survey whether they thought that literature ‘can 
be used effectively to treat some mental health condi¬ 
tions’. 40.0% of respondents said yes, probably’, and a fur¬ 
ther 40.4% said yes, definitely'. 80.4% of Start of Course 
survey respondents, therefore, thought that literature at 
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Figure 2: Graph comparing total number of responses to each course activity with the number of responses in which 
an email address was supplied. 
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Figure 3: Graph showing the percentage of responses to each activity in which an email address was supplied. 


least ‘probably’ could be used in a therapeutic context. 
This percentage increases to 89.9% among those respond¬ 
ents whose email addresses can also be matched to the 
End of Course survey, and to 82.0% among those respond¬ 
ents who volunteered for the post-course online study. 
Any results from in-course research activities, or from a 
post-course online study, which support the hypothesis 
that literature can be therapeutically beneficial, are thus 
generated by a sample of which at least 80% of the partici¬ 
pants already believe in the concept being tested. 


The logic of the placebo effect would suggest that 
Literature and Mental Health learners, whether within 
the course or as part of a post-course study, are likely to 
experience an improvement in wellbeing when they read 
precisely because they expect to experience this improve¬ 
ment. Admittedly, the precise status of the placebo effect in 
non-pharmacological trials is the subject of some debate. 
The placebo effect refers to any change in a patient’s con¬ 
dition, whether positive or negative, which cannot be 
directly attributed to the treatment administered, and 













Art, 19, page 8 of 17 


Hodge: Adapting a MOOC for Research 


must therefore result from other factors: in particular, 
from the patient’s belief in the treatment, but also, per¬ 
haps, from the medical rituals of receiving treatment and 
consulting with a doctor, in pharmacological trials, the 
placebo effect is much easier to isolate. The real drug is 
administered to one group, and the placebo to another, 
and the results in both groups can be observed and com¬ 
pared, to identify which changes are caused by the drug 
itself. With non-pharmacological treatments, however, 
the surrounding factors that cause the placebo effect are 
often much harder to separate from the treatment itself. 
Trials have proven mindfulness to be more effective in 
reducing sensations of pain than both a placebo analge¬ 
sic cream (actually petroleum jelly) and a sham mindful¬ 
ness practice (Zeiden et al., 2015). However, there exists 
an extensive literature discussing the difficulty of distin¬ 
guishing the placebo effect in psychotherapy (see, for 
example, Wampold et al., 2005; Brown, 2006; Justman, 
2011; Wedge, 2012). Several researchers have argued that 
factors which would usually be regarded as contributing 
to the placebo effect, such as a patient’s confidence in 
and hope for the treatment, or the fact of regularly visit¬ 
ing a medical practitioner, are not extraneous in psycho¬ 
therapy; rather, they are an integral part of the treatment 
itself (Justman, 2011; Wedge, 2012). 

The status of the placebo effect for bibliotherapy-based 
treatments has not yet been elucidated by research; we 
do not yet know, in other words, whether a patient’s con¬ 
fidence in bibliotherapy is integral to the outcome of 
their treatment. The self-selection bias within the MOOC 
does not immediately or entirely negate the results of 
the course research activities, therefore, and nor does it 
render an extended online research trial involving MOOC 
participants a futile exercise. The fact that 6707 people 
believe in the therapeutic potential of literature should 
in itself be regarded as a positive finding, as should the 
fact that activity participants drawn from this cohort reg¬ 
istered lower levels of stress after reading or listening to a 
poem. The bias must be acknowledged, however, and will 
inevitably influence the online trial design. 

Self-selection bias will always be a relevant considera¬ 
tion for MOOC researchers. Whereas the participants in 
the mindfulness trials discussed above were selected 
for their suitability, as well as for their willingness to be 
involved in a study, participants in any MOOC research 
activity have all volunteered to learn about a particular 
subject, in a particular way. Whether research is being 
conducted into a subject related to the course content (as 
with Literature and Mental Health), or into the efficacy of 
an online teaching method, the potential influence of the 
participants’ self-selection bias on any findings from the 
research must be acknowledged. The Start of Course sur¬ 
vey results indicate to us the extent of that bias. 

Balancing research and learning 

Cited above, the FutureLearn research ethics guidelines 
describe learners as ‘[ijndividuals... engaging voluntarily 
with the courses with the intention of learning’ (Future¬ 
Learn, 2014). FutureLearn Partners have an ethical respon¬ 
sibility to alert learners to any research being conducted 


on or via a MOOC, but they also have a responsibility, as 
educational providers, to ensure that any research taking 
place does not impinge on or impede the MOOC’s pri¬ 
mary purpose, as a teaching and learning tool. Overall, 
the research activities developed for Literature and Mental 
Health were successfully integrated into the course learn¬ 
ing experience. More than 80% of learners, according to 
the End of Course survey responses, enjoyed taking part in 
the interactive exercises developed for research purposes, 
and the discussion steps inspired by the surveys gener¬ 
ated some of the highest numbers of comments of any 
steps throughout the course. We found that, designed and 
framed in the right way, research activities can enrich the 
learning experience. In Literature and Mental Health, as 
in some other MOOCs, the learners became part of the 
investigative process, exploring the same questions and 
concepts that we wanted to explore through our research. 
In order to prioritise the MOOC’s function as a teaching 
and learning tool, however, some important aspects of a 
standard research trial were sacrificed. Specifically, it was 
not possible to introduce a control into some of the activi¬ 
ties, which therefore diminished the validity of the results. 

When we began developing the research activities for 
Literature and Mental Health, we were conscious of the 
need to minimise the burden placed on learners by any 
surveys or exercises. In the absence of tests or quizzes, the 
three interactive poetry activities (the Poetry for Stress 
Relief activity, the Missing Word activity and the Poetry and 
Memory activity) were in fact designed, first and foremost, 
as opportunities for learners to explore and apply some of 
the ideas set out in the course videos. The Poetry for Stress 
Relief activity, for instance, gave learners the opportunity 
to find out whether reading or listening to a poem could 
make them feel ‘calm’, ‘relaxed’, ‘composed’ and ‘focused’, 
as lead educator Dr Paula Byrne suggests in step 1.5 
(University of Warwick, 2016). Learners were exploring, at 
an individual level, the same question that we wanted to 
explore on a large scale: can reading a short poem help to 
alleviate stress? Other MOOCs have integrated potential 
experiments in a similar way. The University of Warwick’s 
FutureLearn MOOC The Mind is Flat: The Shocking 
Shallowness of Human Psychology invites learners to par¬ 
ticipate in a different psychological experiment in each 
of its six weeks. The learners effectively become research 
subjects, but the process is designed to supplement their 
understanding of the course content; according to step 
1.13, introducing the Week 1 experiment, ‘each experi¬ 
ment illustrates something about how our minds work’ 
and ‘givejs] [learners] a sense of how psychologists use 
experiments to uncover... aspects of the mind' (University 
of Warwick, 2013). 

We see here how research and learning may produc¬ 
tively coexist as functions of a MOOC, at least in theory. 
In Literature and Mental Health, in practice, learners felt 
that they gained less from the interactive research activi¬ 
ties than from other aspects of the course. Figure 4 shows 
learner responses to a question in the End of Course sur¬ 
vey asking which feature of the course they learned most 
from. ‘Interactive activities’ scored least well among the 
options given, although responses to the question ‘Which 
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of the following features of the course did you learn 
least from?’ (Figure 5) indicate that ’online discussions 
with other learners’ were least instructive for the highest 
number of learners. Just under a quarter of respondents 
felt that they learned least from the interactive research 
activities. However, in the same survey, a total of 84.9% 
of learners indicated that they either ‘somewhat’ or ‘defi¬ 
nitely’ enjoyed taking part in these exercises. This would 
suggest that the research activities, although they were 
not as informative as the course educator videos or the 


interviews with writers and GPs, were neither burden¬ 
some to learners, nor detrimental to the overall course 
experience. 

More difficult to integrate into the course learning 
experience were the three surveys. FutureLearn’s research 
ethics guidelines require Partners to be ‘sensitive to the 
problem of inundating learners with surveys.’ In addition 
to the concerns about learning safeguarding discussed 
above, it was our awareness of the need to minimise the 
number of surveys distributed to learners that caused us 
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Figure 4: Graph showing learner responses to the End of Course survey question ‘Which of the following features of 
the course did you learn most from?’ 
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Figure 5: Graph showing learner responses to the End of Course survey question ‘Which of the following features of 
the course did you learn least from?' 
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to leave out the PHQ-9, a validated wellbeing assessment 
used by medical practitioners in the diagnosis of certain 
mental health conditions (Kroenke et al., 2001; Martin et 
al., 2006). All research activities included within MOOC 
steps, moreover, were required to have a learning outcome. 
In the step immediately following each of the surveys, 
therefore, we invited learners to reflect on and, if they felt 
comfortable doing so, to discuss their survey responses. 
After the Start of Course survey, learners were encouraged 
to consider whether they had realised something new 
about their reading habits from taking part in the survey, 
and whether their responses would influence their read¬ 
ing habits. The Reading for Wellbeing survey was followed 
by a discussion of which books learners read for comfort, 
while the End of Course survey was used as the stimulus 
for learners to share their reflections on the course. 

Other MOOCs have also encouraged the use of research 
surveys for learner self-reflection, though in some 
slightly different ways. The first presentation of Monash 
University’s FutureLearn MOOC Mindfulness for Wellbeing 
and Peak Performance, for example, gave learners the 
opportunity to fill in a ‘stress survey’ in its first and final 
weeks. Learners were invited to use the survey to ‘moni¬ 
tor [their] progress through the course and measure the 
impact of mindfulness on [their] life’, and were encour¬ 
aged to print a copy of their responses in order to make it 
easier to compare their results before and after taking the 
course. The inclusion of these results in a research study 
was optional; learners were able to fill in the survey for 
their own records without making their responses avail¬ 
able to the researchers (Monash University, 2015). 


We see here that the relationship between research 
and learning need not be an antagonistic one; the two 
can exist, in practice, as mutually beneficial, rather than 
mutually exclusive processes. That the research surveys 
included in Literature and Mental Health were beneficial 
rather than detrimental to the learning experience is indi¬ 
cated by the popularity of the related discussion steps. 
The opportunities for reflection afforded by the three in- 
MOOC surveys generated some of the highest numbers 
of comments of any steps throughout the course. Indeed, 
the discussion step following the Start of Course survey, 
which invited learners to reflect on and discuss their read¬ 
ing habits in the light of their survey responses, generated 
3717 comments, over 400 comments more than any other 
course step. This volume of responses may be attributed, 
in part, to the fact that this was the first discussion step of 
the course; likewise, the discussion step following the End 
of Course survey may have generated more comments (a 
total of 1071) than any other step in week 6 because it 
is the final discussion step of the course. By contrast, the 
discussion step after the Reading for Wellbeing survey in 
week 5 generated only the third highest number of com¬ 
ments for that week, with 958 comments. 

Figure 6 compares the number of comments gener¬ 
ated by each of the survey discussions to the number of 
comments generated by other discussion steps in the 
same week, as well as to the average number of comments 
across all steps in the same week. Although the Reading 
for Wellbeing survey inspired less discussion than the Start 
and End of Course surveys, it is evident that there was a 
high level of engagement in all three of the discussion 
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Figure 6: Graph comparing the numbers of comments across all discussion steps during the first presentation of Lit¬ 
erature and Mental Health: Reading for Wellbeing. 
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steps following the course surveys. Each one received a 
higher than average number of comments not only for the 
week in which it featured, as Figure 6 demonstrates, but 
also for the course as a whole: the mean number of com¬ 
ments per step for the course as in its entirety is 936.4. 
The surveys had initially posed a greater challenge when 
it came to integrating research activities into the course. 
Whereas the interactive exercises were learning experi¬ 
ences in themselves, the surveys required additional steps 
in order to generate a learning outcome for participants. 
Ultimately, however, the surveys were integrated into the 
course learning experience as successfully as the interac¬ 
tive exercises, if not more so. The evident popularity of the 
discussion steps following the surveys suggests that the 
surveys gave learners food for thought: that learners had 
more to say after reflecting on and answering the survey 
questions. Thus, we can conclude that the surveys were 
beneficial to the course learning experience, encouraging 
learners to think and fostering conversation. 

While we did not find research to be detrimental to 
learning in the MOOC, the necessary prioritising of the 
latter did, at times, inhibit the success of the former. In 
particular, we were not able to introduce a control into 
the course research activities. This had detrimental impli¬ 
cations for the Poetry for Stress relief activity. Participants 
were asked to assess their current mood on six sliding 
scales, before and after either listening to or reading a 
poem. The six scales are set out below: 


Focused on one thing 

Distracted by lots of things 

Physically relaxed 

Physically tense 

Mentally calm 

Mentally anxious 

At ease, able to cope 

Overwhelmed 

Hopeful 

Hopeless 

Uplifted, upbeat 

Unhappy, down 


In our analysis of participants’ responses to this activity, 
we found that there was an average reduction in stress 
levels - that is, a movement towards the left on the scales 
set out above - on every scale, for every version of the 
poem. On average, learners measured themselves to be 
more focused, more relaxed, calmer, more able to cope, 
more hopeful and more uplifted after reading or listening 
to W. H. Davies’ ‘Leisure’. 

This has the potential to be an overwhelmingly positive 
result, indicating that reading and listening to poetry does 
help to reduce stress. Because our participants were learn¬ 
ers in a literature MOOC rather than research subjects, 
however, we did not feel that it would be appropriate to 
create a control group within any of the course activities. 
Instead of reading or listening to a poem, a control group 
in the Poetry for Stress Relief activity would have spent an 
equivalent amount of time listening to a piece of music or 
looking at a photograph. Through comparison, it would 
have then been possible to begin to establish whether 
poetry was uniquely effective at reducing stress. Without 
a control group, however, we cannot be sure that the 


reductions in stress registered by learners in the Poetry 
for Stress relief activity were caused by the specific act of 
reading or listening to a poem. Any act of prolonged con¬ 
centration on a single stimulus may have produced simi¬ 
lar results. 

It would, of course, have been both impractical and 
unfair to make some learners listen to a piece of music or 
look at a still image in order to form a control group in the 
Poetry for Stress Relief activity. This is one of the reasons 
why it is important for a post-course study to take place, 
in a post-course online study, we would not have the same 
responsibility to ensure that all participants receive an 
equivalent learning experience, and have the opportunity 
to take part in a literary exercise. It would therefore be 
possible to introduce control activities to provide a useful 
point of comparison. 

Impact findings 

When we began designing the research activities to be 
included in Literature and Mental Health, we were clear 
that our interest was in using the MOOC to find out 
about the relationship between reading and wellbeing. 
The course was the platform and the vehicle only: our 
research was not intended to find out about MOOCs. 
Reflecting on the outcomes of our research, however, it is 
apparent that we have made some notable findings about 
the impact of the MOOC. Indeed, our impact findings are 
perhaps the most substantial. For the reasons outlined 
above - namely, the problem of self-selection bias and the 
necessity of balancing research and learning - it has not 
been possible, through the MOOC alone, to definitively 
prove a link between reading and wellbeing. From the 
results of the follow-up survey, however, we can see that 
the MOOC itself has had an impact on learners’ reading 
and online learning habits. 

Of the respondents who had not viewed the course for 
two months or more, 27.2% indicated that they were read¬ 
ing more regularly as a result of the course, and 41.2% 
said that they were reading different kinds of texts hav¬ 
ing taken part in the MOOC. Of that 41.2%, 87.7% had 
been encouraged to read more poetry; this is important 
because ReLit places particular emphasis on the read¬ 
ing of poetry, as the Stressed Unstressed poetry anthol¬ 
ogy demonstrates (Bate et al., 2016). Among respondents 
who had not viewed the course for at least one month, 
the figures were slightly higher, with 27.8% reading more 
regularly, and 45.0% reading different kinds of texts. This 
perhaps suggests that the impact of the course dimin¬ 
ishes over time, as the percentage of respondents whose 
reading habits have changed is smaller among those who 
last viewed the course over two months ago, compared to 
those who viewed the course over one month ago. Further 
surveys would be needed to confirm this trend. 

As well as asking about reading habits, the follow-up 
survey also asked respondents about their online learning 
habits. Of learners for whom Literature and Mental Health 
was their first experience of online learning, a substantial 
majority of 70.2% had already signed up for other MOOCs 
by the time the follow-up survey was distributed. This is a 
particularly encouraging figure, suggesting that Literature 
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and Mental Health , as a free online course, gave learners a 
positive experience that they were keen to repeat. The fol¬ 
low-up survey also indicated slight changes in social learn¬ 
ing habits. Of the learners who responded, 493 indicated 
that they ‘sometimes’ or ‘often’ made comments during 
online courses before taking part in Literature and Mental 
Health] this figure had risen to 525, based on the same 
group of respondents, after taking part in Literature and 
Mental Health. 

Having matched responses from the Start of Course and 
End of Course surveys, we can also measure the impact 
of the course by looking at the changes in response to 
certain questions. Based on the same cohort of respond¬ 
ents, answers to the question ‘do you think literature can 
be used effectively to treat some mental health condi¬ 
tions?’ in the Start and End of Course surveys indicate that 
the course itself changed learners’ attitudes to ‘reading 
for wellbeing’. The number of learners responding yes, 
definitely’ to the question rose from 434 at the start of 
the course, to 475 by the end. Moreover, the number of 
learners indicating that they were ‘undecided’ more than 
halved, dropping from 141 to 64. Although there was an 
increase, from 4 to 22, in learners responding ‘no, prob¬ 
ably not’, therefore, these figures do indicate that the 
course was effective in encouraging learners to formulate 
definite opinions about the relationship between reading 
and wellbeing. 

These figures correspond with the results from the fol¬ 
low-up survey, indicating that more learners were reading 
for comfort and reading for stress-relief as a result of tak¬ 
ing part in the course. The number of learners reading for 
comfort ‘often’ increased by almost 100, from 313 to 410; 
the number reading for comfort ‘rarely’ or ‘never’, mean¬ 
while, dropped from 212 to 128. Similarly, the number of 
learners reading for stress-relief‘often’ increased from 273 
to 399, while the number reading for stress-relief ‘rarely’ 
or ‘never’ dropped 273 to 151. 

At most, of course, these figures tell us about the impact 
of the course ten weeks after its conclusion. 40.25% of 
respondents to the survey (506 people in total) had not 
viewed the MOOC for over two months, but the remain¬ 
der of respondents were completing the survey within 
eight weeks of last visiting the course. We can conclude 
very little, therefore, about the long-term, lasting impact 
of Literature and Mental Health. The fact that changes 
in people’s reading habits have been sustained for two- 
month period does not mean that learners’ behaviour will 
still be influenced by the course six months or one year 
later. Nonetheless, the results of the follow-up survey do 
demonstrate that Literature and Mental Health has had an 
impact upon the habits and attitudes of its learners. We 
should be encouraged by these results, and consider seek¬ 
ing further follow-up opportunities once more time has 
elapsed. 

The value of qualitative data 

It was the big numbers that made MOOC research seem an 
attractive prospect at the project’s outset: the opportunity 
to observe mass trends, and the potential for thousands of 
participants to respond positively to a bibliotherapy prac¬ 


tice. We can learn much from the numerical data, as with 
the results of the follow-up survey discussed above. We 
had not realised, however, how much more there would 
be to learn from the qualitative data gathered during the 
course: from the stories of individual learners as opposed 
to mass statistical trends. Indeed, we had not anticipated 
how willing learners would be to supply this qualitative 
data, to respond to optional open-ended questions in the 
surveys, to make comments in the course discussions and 
to recount their personal experiences. Figure 7 shows the 
number of responses to each of the course research activi¬ 
ties and, where available, the number of those responses 
which included at least one answer to an open-ended 
question. None of these open-ended questions was com¬ 
pulsory. From the graph, it is clear that the majority of 
responses to both the End of Course and the Follow-Up 
survey contained at least one answer to an open-ended 
question. 76.81% of End of Course survey responses, and 
65.84% of Follow-Up survey responses contained at least 
one of these answers. The percentage for the Reading 
for Wellbeing survey, meanwhile, was 47.73%, with just 
under half of the total responses offering an answer to an 
open-ended question. It should be noted that the notice¬ 
ably smaller percentage for the Start of Course survey may 
be due, in part, to the fact that the only open-ended ques¬ 
tions were the ‘Other, please specify’ options in any drop¬ 
down lists, and a question asking anyone who has been 
employed in the medical or care professions to describe 
their role. Examples of the open-ended questions from 
the other three surveys, by contrast, include the following: 

• ‘If you feel comfortable doing so, we would also be 
very interested to know how and why you feel that 
this piece of literature changed your life'( Reading for 
Wellbeing survey) 

• ‘Do you have any final comments, suggestions or ob¬ 
servations to make about the course?’ (End of Course 
survey) 

• ‘Do you have any further comments about how, if at 
all, Literature and Mental Health has changed your 
reading habits?’ (Follow-Up survey) 

We had been conscious of burdening learners with open- 
ended questions such as these when we designed the sur¬ 
veys, including only a limited number and ensuring that 
all were optional. We were both surprised and pleased, 
therefore, to find that so many respondents were willing 
to engage with these questions. This was a MOOC in which 
many learners wanted to tell their stories, and to supply a 
descriptive or narrative response to a question rather than 
simply ticking a box. We had invited learners to become 
part of a conversation (University of Warwick, 2016), and 
this ethos of conversation was adopted by many learners 
in their survey responses. 

The sheer volume of comments made during the course is 
further testament to the learners’ willingness to engage in 
a conversation, and to offer reflective, qualitative feedback 
in response to the course materials. Figure 8 compares 
the total number of comments made across all presenta¬ 
tions of the University of Warwick’s five existing MOOCs. 
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Figure 7: Graph comparing total number of survey/activity responses to the number of survey/activity responses con¬ 
taining at least one answer to an open-ended question. 
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Figure 8: Graph comparing the total number of comments made across all presentations of the University of Warwick’s 
five existing MOOCs. 

SaHW: Shakespeare and His World. 

TMiF: The Mind is Flat: The Shocking Shallowness of Human Psychology. 

BD: Big Data: Measuring and Predicting Human Behaviour. 

BiM: Babies in Mind: Why the Parent’s Mind Matters. 

LaMH: Literature and Mental Health: Reading for Wellbeing. 
























Art, 19, page 14 of 17 


Hodge: Adapting a MOOC for Research 


In its first presentation, Literature and Mental Health 
received almost twice as many comments as any other 
Warwick MOOC; it received more comments than other 
Warwick MOOC running at a similar time, and it received 
more comments than any other Warwick MOOC running 
for the first time. These higher comment numbers can, in 
part, be attributed to higher learner numbers. Literature 
and Mental Health is the only MOOC included in the graph 
to have over 10,000 learners, with a total of 12,485 learn¬ 
ers taking part. It is no coincidence that the MOOC that 
generated the second highest number of comments - the 
second presentation of‘Shakespeare and His World’ - also 
had the second highest number of learners, at 9,341. 

Even allowing for the discrepancies in learner numbers, 
however, the comment total for Literature and Mental 
Health remains comparatively high. Figure 9 shows the 
average number of comments per learner for each presen¬ 
tation of the University of Warwick’s five MOOCs. We see 
here that the first presentation of Literature and Mental 
Health generated more than twice as many comments per 
learner as any presentation of The Mind is Flat, Big Data 
or Babies in Mind. It also generated more comments per 
learner than any run of Shakespeare and His World, by 
more than 1.2 comments. 

Moreover, Literature and Mental Health had a higher 
percentage of social learners in its first presentation than 


any other presentation of a University of Warwick MOOC 
to date. 41.65% of learners made at least one comment. 
The first presentation of Babies in Mind came closest to 
this social learner figure, with 39.29% of learners becom¬ 
ing social learners. Social learner figures for presentations 
of Shakespeare and His World, meanwhile, ranged from 
25.82% to 32.43%; fewer than 25% of learners made 
at least one comment in any presentation of The Mind 
is Flat for which the required records are available, and 
just under 19% of learners became social learners in both 
presentations of Big Data. 

A greater proportion of the Literature and Mental Health 
learners chose to make at least one comment, to join the 
conversation. If we compare Literature and Mental Health 
with the most recent run of Shakespeare and His World 
(18 th April 2016), we see that the average number of com¬ 
ments per social learner for Shakespeare and His World is 
actually higher than for Literature and Mental Health, at 
19.45 compared to 15.87. However, the overall percentage 
of social learners is smaller by approximately one third: 
26.04% of Shakespeare learners became social learners, 
compared to 41.65% of learners on Literature and Mental 
Health. In Shakespeare and His World, fewer learners 
were making more comments, whereas in Literature and 
Mental Health, a greater proportion of learners felt able to 
contribute to the discussions. 
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Figure 9: Graph showing average number of comments per learner for each presentation of the University of Warwick’s 
five existing MOOCs. 

SaHW: Shakespeare and His World. 

TMiF: The Mind is Flat: The Shocking Shallowness of Human Psychology. 

BD: Big Data: Measuring and Predicting Human Behaviour. 

BiM: Babies in Mind: Why the Parent’s Mind Matters. 

LaMH: Literature and Mental Health: Reading for Wellbeing. 
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It is apparent that the learners who participated in the 
first presentation of ‘Literature and Mental Health’ were 
extremely and unusually willing to supply qualitative data, 
and that we consequently have a large amount of qualita¬ 
tive data to consider. While we have not begun any for¬ 
mal process of semantic analysis, we have already found 
learner comments and open-ended survey responses to 
be of considerable value, particularly in establishing the 
impact of the MOOC. These individual narratives are a use¬ 
ful and illuminating counterpart to the mass trends iden¬ 
tifiable from the qualitative data. 

Given the abundance of qualitative data available, this 
paper will limit itself to responses to three open-ended 
questions from the Follow-Up survey: 

• Do you have any further comments about how, if at 
all, Literature and Mental Health has changed your 
attitude to online learning? 

■ Do you have any further comments about how, if at all, 
Literature and Mental Health has changed the types of 
text your read? 

■ Do you have any further comments about how, if at all, 
Literature and Mental Health has changed your read¬ 
ing habits? 

Several responses to these survey questions describe per¬ 
sonal improvements in wellbeing brought about by the 
course. One participant wrote of how Literature and Mental 
Health had introduced them to poetry-reading as a form 
of stress relief, and described how they were now reading 
poetry aloud when they felt anxious, or when they were 
unable to sleep. A number of other respondents said that 
the course had helped them to better understand and 
manage a particular diagnosed condition, including major 
depressive disorder, post-traumatic stress disorder and mild 
cognitive impairment. For others, the course was a source 
of comfort during a difficult time, for example, following 
the death of a parent or during the illness of a loved one. 

Such anecdotal evidence about the impact of Literature 
and Mental Health is powerful, telling us, in specific and 
human terms, how the course has changed the lives of 
some of the individuals who participated. Moreover, the 
responses to the Follow-Up survey’s open-ended ques¬ 
tions also offer us some insight into the broader reach of 
the course. Several respondents described how they were 
now passing on the message of the course to others, both 
personally and professionally. Some stated that they were 
using what they had learned on the course to help loved 
ones, who were suffering either with stress or with the 
early stages of dementia. Others, meanwhile, described 
how they had introduced some elements of the course into 
a professional setting. One respondent had begun incor¬ 
porating poetry into therapy sessions, one had distributed 
some poems in a local clinic waiting room, and another 
had created a mental health and wellbeing information 
board in the library of the school in which they work. 
In some instances, the actions described will have made 
ideas about reading and wellbeing available to hundreds 
of other people. Poems distributed in waiting rooms, and 
mental health and wellbeing boards established in school 


libraries, encourage people who may not have taken part 
in Literature and Mental Health to consider the therapeu¬ 
tic potential of reading. Public display boards raise aware¬ 
ness and start conversations about mental health and 
wellbeing. 

The qualitative data gathered during Literature and 
Mental Health, then, offers us variety, subjectivity, the indi¬ 
vidual experience; but it can also hint, like the quantita¬ 
tive data, towards the big numbers. From the Follow-Up 
survey responses, we have discovered the course's impact 
to be more far-reaching than we would otherwise have 
anticipated. Much MOOC-related research already incor¬ 
porates semantic analyses of learner comments (see, for 
example, de Ward and Kukulska-Hulme, 2016; Hudson, 
2016) and going forwards, we will consider a more rig¬ 
orous and systematic approach to the analysis of open- 
ended responses from Literature and Mental Health. Given 
that learners did not seem burdened by open-ended ques¬ 
tions, we feel that future MOOC research projects could 
benefit from providing more opportunities for this kind 
of data gathering. 

Conclusion 

Our experience of researching via Literature and Mental 
Health has been, overall, an instructive and encourag¬ 
ing one. We have found research to be compatible with 
teaching and learning within a MOOC, to the extent 
that research activities can actually enhance the learn¬ 
ing experience. Given the resources expended during the 
production of a MOOC, therefore, it seems sensible for 
universities to harness Massive Open Online Courses as 
opportunities for research as well as for teaching wher¬ 
ever possible. Indeed, the ability of universities to capital¬ 
ize on these research opportunities may be crucial to the 
future of the MOOC, as the MOOC business model comes 
under increasing scrutiny. 

Researchers are already thinking creatively about MOOC 
research, and are already thinking beyond the MOOC 
itself, into subject areas (as with Mindfulness for Wellbeing 
and Peak Performance and Literature and Mental Health) 
and into other online teaching and learning platforms 
(see, for example, O'Riordan et al., 2015, on measures of 
online pedagogical activity). As more researchers begin to 
think through and with MOOCs, we would advise a greater 
emphasis on the gathering and analysis of qualitative data, 
which the Massive Open Online Course, in our experience, 
seems particularly suited to generating. 

Our first attempt at researching through Literature and 
Mental Health has identified some notable weaknesses of 
the MOOC as a research tool. The self-selection bias that 
exists among a cohort of research participants drawn from 
a MOOC is both inevitable and ineradicable. The relative 
anonymity of learners also posed a problem for us, and 
may for other MOOC researchers, because of the difficulty 
of linking up responses made, by the same individual, to 
separate research activities. Nevertheless, we have found 
the MOOC platform to be surprisingly versatile: it can be 
adapted for research, thereby providing researchers with 
ready access to thousands of possible research partici¬ 
pants. If more researchers begin to take advantage of this 
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opportunity, then we may witness a new, research-driven 

era in the ongoing evolution of the MOOC. 
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